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In an economy that increasingly demands postsecondary credentials to access high-paying jobs with the potential 
for career advancement, students need comprehensive and reliable information to make better college and career 
choices. This need has prompted a vigorous national dialogue about the best way to provide information on the 
performance of postsecondary institutions, as well as how this information should be used to inform students and 
encourage improved outcomes. The U.S. Department of Education’s Postsecondary Institution Rating System 
(PIRS) proposal, along with consumer reporting provisions included in legislation to reauthorize the Higher 
Education Act, advances this dialogue and sets the stage for prospective solutions. As these efforts move forward, 
policymakers at the federal and state levels should give special attention to the needs of students from 
disadvantaged backgrounds. Substantial improvements in the availability of consumer information, as well as the 
eventual use of this information for greater accountability, are possible, but careful design of these information 
systems is essential to minimize the risk to the nation’s most vulnerable students. 

While recent Department of Education initiatives, such as the College Scorecard, have greatly strengthened the 
information base available to the public, there is more work to be done to provide better and more comprehensive 
information about access, progress, completion, and important post-graduation outcomes. This is the impetus for 
the Department of Education’s PIRS proposal: 

The Department intends, through these ratings, to compare colleges with similar missions and identify colleges that do 
the most to help students from disadvantaged and underrepresented backgrounds, as well as colleges that are improving 
their performance. The ratings system is not intended to rank institutions. Instead, it will provide information about an 
institution's performance on a specific measure or a specific set of measures. In the upcoming reauthorization of the 
Higher Education Act, the President will propose allocating financial aid based upon these college ratings by 2018. 1 

CLASP supports increasing transparency and accountability for postsecondary results and sees PIRS as an 
important instrument to promote these goals. This paper presents recommendations for implementing PIRS in a 
way that supports the goal of empowering students by providing the information they need to make informed 
decisions about their postsecondary plans while also avoiding unintended consequences, especially for students 
from disadvantaged backgrounds, including low-income and under-represented students. The paper is based on 


1 Request for Information To Gather Technical Expertise Pertaining to Data Elements, Metrics, Data Collection, Weighting, Scoring, and Presentation of a 
Postsecondary Institution Ratings System 
A Notice by the Education Department on 12/17/2013. 
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CLASP’s PIRS comments to the Department of Education, as well as our testimony to the Advisory Committee on 
Student Financial Assistance on PIRS. CLASP has also authored a companion briefing paper, Workforce Results 
Matter: The Critical Role of Employment Outcome Data in Improving Transparency of Postsecondary Education 
and Training , focusing on the importance of including employment-related outcomes, such as post-graduation 
employment rates and earnings levels. 

Transparency vs. Accountability 

Our recommendations for PIRS recognize the sharp distinction between the goal of increasing transparency of 
information about postsecondary education and the goal of holding institutions or programs accountable for 
outcomes. Transparency in this context refers to the ability of postsecondary education consumers to access the 
facts they need to make an informed decision about whether to enroll in postsecondary education, what to study, 
where to enroll, and how to finance their education. Accountability can take many forms, including the creation and 
imposition of minimal standards of performance that are meant to remove unqualified providers. It can also involve 
financial or other incentives to spur performance improvements or corrective actions intended to help institutions 
meet minimal standards. Accountability is concerned not only with what the results are for an institution but also 
what they should be (i.e., what is reasonable and fair to expect). 

Any foreseeable use of PIRS may have unintended consequences that should be minimized in the ways this paper 
suggests. Using PIRS to support institutional accountability, in particular, creates special concerns about the 
potential for unintended consequences. The stakes would be very high; PIRS could affect an institution’s Title IV 
eligibility or the amount of Title IV funds available to an institution. Using PIRS for institutional accountability 
would be much more likely to lead to undesired responses than would other uses. It is reasonable to imagine that, 
confronted with the prospect of losing funds, institutions might reduce their focus on Pell grant recipients and other 
lower-income students or otherwise change their enrollment process in a way that reduces opportunities for lower- 
income students. 

In our view, it is possible to implement PIRS in a way that greatly reduces the potential for these types of negative 
responses. The potential pitfalls are real; however, the solution is not to forego establishing PIRS but rather to 
develop and use PIRS the right way. This paper outlines several recommendations that are intended to accomplish 
this. Whatever form PIRS ultimately takes, we recommend that its design and uses be carefully assessed to estimate 
the potential impact on students from disadvantaged backgrounds, including low-income and under-represented 
students. For more on this, see Reforming Student Aid: How to Simplify Tax Aid and Use Performance Metrics to 
Improve College Choices and Completion. 

Recommendation 1: PIRS should not be used for accountability purposes until 
several critical data shortcomings have been addressed. 

Existing data sources may be adequate to provide consumer information. However, these data sources are in need 
of substantial improvement . 2 For this reason, we do not support the use of PIRS for accountability purposes until 
several critical data shortcomings are addressed, including additional data collection and improved connections 


2 Mapping the Postsecondary Data Domain: Problems and Possibilities, Technical Report. Mamie Voight, Alegnetta A. Long, Mark Huelsman, and Jennifer 
Engle. March 2014, Institute for Higher Education Policy, Washington, DC. 
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between federal databases that are needed to calculate several of the proposed metrics. Key data needs include: 

• Increased coverage of students: Completion rates are difficult to estimate because of limited data coverage 
of students in community colleges and other institutions that serve students who are not attending school 
for the first time. This is beginning to change with the collection of additional information on non-first time 
and part-time students through the Integrated Postsecondary Education Data System (IPEDS). The 
expected changes to IPEDS will increase the coverage of non-first-time and part-time students and should 
eventually allow for calculation of an expanded graduation rate. 

• More information on student characteristics to support calculation and disaggregation of outcomes for 
sub-groups of students: Student outcomes available through IPEDS cannot be disaggregated by enrollment 
status, age, socioeconomic status, or other student characteristics that can provide a complete picture of 
postsecondary results. For example, the revised version of the IPEDS outcome measures are expected to be 
reported in the aggregate for all students in an enrollment category. Ultimately, student-level data may need 
to be made available to ensure access and outcome data include the disaggregation needed for transparency 
and accountability. 

• Additional data on student progress, especially in less selective institutions: The measures of enrollment in 
remedial instruction and enrollment in college-level instruction cannot be calculated without new data 
collection. These measures, as outlined below in recommendation 2, are needed to assess the extent to 
which colleges are helping students progress toward graduation. Here, as above, access to student-level 
data may be necessary to efficiently gather this information. 

• Access to reliable information on post-program earnings. Although earnings measures can be calculated 
now for a limited set of students in the National Student Loan System Database (NSLDS), substantial 
improvements are needed to make broader use of this data. Access to national or cross-state employment 
data is needed to provide information about labor market outcomes, such as median earnings. 3 In addition, 
these data must be connected to information about the student’s program of study, so that earnings results 
can be produced by program. Otherwise, earnings comparisons between institutions will simply reflect their 
varied program offerings. 


Recommendation 2: PIRS should incorporate a range of metrics, including measures 
of student progress, and should avoid reducing these results to a composite rating 
of institutions for transparency or accountability purposes. 

CLASP recommends including a range of measures that reflect the goals of student access, student progress, 
completion, and post-graduation outcomes (Table 1). The advantage of a range of measures is that it better reflects 
the array of key goals for postsecondary institutions, including student access and progress, as well as completion 
and post-graduation outcomes. This will help reduce the focus on a single objective, such as graduation rate, at the 
expense of other goals. 


3 Workforce Results Matter: The Critical Role of Employment Outcome Data in Improving Transparency of Postsecondary Education and Training. Tim 
Harmon, Neil Ridley, with Rachel Zinn, Workforce Data Quality Campaign, April 2014, CLASP, Washington, DC. 
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Table 1. Examples of PIRS Measures 


Categories and Focus Examples of Measures 


Access and affordability: College access and 
college costs. Does this institution provide access 
to low-income students? Is it affordable for 
students from a range of backgrounds? 

Percent of students receiving Pell or other need-based financial 
aid 

Net price 

Student Progress: Progress of students toward 
completion. Are students at this institution 
showing progress toward completing their 
education by reaching key milestones? 

Percent of students retained in subsequent year 

Percent of students enrolled in and completing developmental 

Percent of students completing college-level “gateway” courses 

Credit accumulation in a postsecondary program of study 

Completion: Student completion and attainment 
of credentials. How many students attain a 
credential or degree from this institution? How 
many transfer to another college? 

Certificate/credential attainment rate 

Degree attainment rate 

Transfer rate 

Post-graduation results: Employment, earnings, 
and debt burden after graduation. What levels of 
earnings do students have following graduation? 
Are students able to repay their loans? Are they 
burdened with high debt? 

Employment rate 

Median earnings one to two years following graduation 

Median earnings five years following graduation 

Average debt of students with college loan debt 

Loan repayment rates 


Consistent with the recommendation to include a range of metrics, CLASP also recommends that the results of 
these metrics not be reduced to a single composite rating or score. Because there are so many factors that contribute 
to institutions’ composite ratings, their value to consumers is very limited. Not all metrics are of equal importance 
to consumers. Some may, for example, place a higher priority on net price and a lower priority on graduation rate. 
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Composite ratings could be especially problematic if PIRS is used in the future for accountability purposes; they 
may be difficult to understand and difficult to use for program or institutional improvement. Treating institutions 
fairly would seem to require that any disqualification thresholds be based on clear criteria. These criteria should be 
designed to remove those institutions that fail to meet minimal standards of performance over time. Setting 
thresholds based on a single criterion can create strong incentives to game results or distort institutional missions 
and may also have other undesired results. No institution should ever be confused about the minimal threshold or 
the metric(s) they must improve to protect Title IV eligibility. 

Recommendation 3: PIRS should empower students to use information effectively 
by disaggregating results for key student subgroups and by supporting the use of 
PIRS data in college and career guidance services. 

CLASP strongly supports using PIRS to promote consumer awareness and choice. Increased transparency can make 
the postsecondary education and training market more functional. Armed with better data, consumers will have 
more options and increased chances of making informed choices about college and career goals. Additionally, 
enrolled students may use information about results at their institution to advocate for institutional improvement. 
Among employers, policymakers, and other stakeholders, consumer information may be used to identify 
achievement gaps and begin to address them. In order to realize this vision in a way that minimizes unintended 
consequences, PIRS should be integrated into college access programs and career guidance, as well as empower 
students to use data effectively. 

Presentation of PIRS results: It is important that metrics results be available, easy to use, and presented to 
students in an appropriate context to support effective decision making. In general, data for each of the metrics 
reported in PIRS should be presented in four ways: 

• Overall results for each metric (How did this college perform on each metric?); 

• Results for subcategories of interest to the consumer , such as programs of study or types of students (How 
did this college perform for students like me?); 

• Comparison of results to the average for institutions in the peer group (How does this college compare to 
other similar colleges?); and 

• Comparison of results to those of the other institutions selected by the consumer (How does this college 
compare to other colleges I have selected?) 

Disaggregation of PIRS results: In order to enable consumers to use PIRS data successfully, information on 
results — not just access — must be presented for sub-groups of students for comparison purposes. This is true for 
two reasons. First, for consumer information purposes, students need to be able to see how each institution 
performs — not just for all students but for students like themselves. A prospective low-income student should be 
able to see institutional results for Pell grant recipients, because these can reveal important differences between 
institutions that would not otherwise be apparent (Table 2). 

For example, if results are not presented for sub-groups for comparison purposes, a Pell-eligible prospective student 
may select College D in Table 2 because of its high overall graduation rate. If that student had access to graduation 
rates for Pell students in particular, he or she might have chosen College C, which has the highest Pell student 
graduation rate. 
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Table 2. Examples of Disaggregation of PIRS Results (Hypothetical) 

Institution 

Graduation Rate 

Percent 
Receiving Pell 
Grant 

Pell Student 
Graduation Rate 

Non-Pell Student 
Graduation Rate 

College A 

45% 

50% 

43% 

47% 

College B 

50% 

35% 

41% 

55% 

College C 

70% 

25% 

60% 

73% 

College D 

75% 

15% 

50% 

79% 


Second, in order for PIRS to be used for accountability purposes, it must show results for categories of students 
from disadvantaged backgrounds. Disaggregating outcomes may help reduce negative consequences for these 
students in the face of increased performance pressure. If results are presented for sub-groups of students together 
with information about access, it becomes possible to measure and begin to address critical achievement gaps or 
disparities in educational and labor market outcomes. Also, these data can be used to “level the playing field” 
between schools that have similar missions but serve very different student populations. 

Student results should be disaggregated and presented for the following sub-groups: 

• Programs of study: This would be helpful for each of the metrics but is of particular importance for 
employment and earnings results, which are most useful when presented in the context of a program of 
study. 

• Pell Grant recipients: Does the institution obtain good results for both Pell recipients and non-Pell 
recipients? 

• Full-time/part-time/mixed enrollment status: Students’ ability to attend full time heavily affects their 
prospects for graduation. It’s a disservice to students not to make this reality clear. Further, students should 
be aware that some schools are more successful than others with part-time students. 

• Gender: Showing institutional results for students by gender, particularly in settings where they will be 
underrepresented, is an important part of the context. 

• Race/ethnicity: The ability of the institution to minimize achievement gaps for minority students is an 
important element for comparison. 

Integrate PIRS data into college access programs and career guidance. A PIRS with results for the suggested 
metrics, disaggregated for key sub-groups, and presented in the ways described above would substantially improve 
the transparency of higher education outcomes. However, it is clear that information alone — even if presented in an 
engaging, easy-to-use format — is not enough. Information on results should be integrated into college access 
programs, as well as the career guidance and college choice delivery systems that reach students at different levels. 
Counselors and advisers should be equipped to interpret this information for prospective and enrolled students. 
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Students should have access to online tutorials or other resources that teach them to find and interpret results for 
institutions and programs. The Department should provide guidance and invest resources to support this function. 

Recommendation 4: PIRS should compare institutions and programs fairly by 
presenting results in comparison to peer institutions and by adjusting expected 
results for student characteristics if used for accountability. 

Compare within Peer Groups. The proposed PIRS should include a process whereby results for institutions are 
presented within peer groups, so that only broadly similar institutions are being compared. In developing these peer 
(or comparison) groups, it is important to distinguish between institutional characteristics that are part of a 
consumer frame of reference (such as location, cost, or size) and institutional characteristics that are inherently 
connected to or that influence the results (such as degree of selectivity or types of credentials granted). For instance, 
it may be entirely appropriate to compare results for institutions of differing location, cost, and size; however, it 
may not be appropriate to compare results for selective and non-selective institutions. The peer groups should 
encourage students to compare institutions that are similar on these fundamental factors. 

Accordingly, consumers should have access to two types of comparative information: 1) information on location, 
cost, and other factors; and 2) information about results based on key institutional differences that influence 
outcomes. Institutions should be grouped along dimensions that have strong predictive power for the metrics. These 
dimensions may include: 

• Level of selectivity (e.g., percent of applicants admitted); 

• Types and levels of credentials granted (e.g., awards, certificates, associates, bachelors, advanced); 

• Percent of students receiving Pell grants or other need-based aid; and 

• Percent of students attending other than full time. 

In addition to the peer group characteristics listed above, which are relevant for both transparency and 
accountability uses of PIRS, there are several additional characteristics of institutions and students that should be 
considered if PIRS is to be used to support accountability. Some examples include: 

• Percent of students enrolled in remedial instruction; 

• Percent of students over 22 years of age at first enrollment or age 24 and older; and 

• Percent of students who are first-generation college students. 

Adjust for Institutional and Student Characteristics. For consumer information purposes, unadjusted data 
should be provided to students and stakeholders. Consumers should be able to use selection criteria to compare 
institutions based on location and other factors. They should also be able to compare these results with those of peer 
institutions. 

Data used for accountability purposes should be treated differently than data used for consumer information. If 
accountability uses of PIRS will include anything beyond setting low minimum thresholds of performance for 
certain metrics that all institutions are expected to meet, then PIRS must have some process for setting institutional 
expectations that takes into account the differences in critical institutional and student characteristics. Without 
incorporating such protections for institutions that enroll low-income students and help them succeed, an 
accountability system will create incentives to enroll and focus resources on the most prepared students and those 
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most likely to succeed in postsecondary education and the job market . 4 

Determining how to do this effectively will be difficult and take time. The advantages and disadvantages of using 
an adjustment model should be carefully weighed. An index or model can be developed to take into account and 
adjust for identifiable characteristics of the students that may influence programmatic or institutional outcomes. 
Regression-based models have been used for years in the calculation of workforce program results to take into 
account different economic conditions and differences in those served. However, an adjustment model is only as 
good as the underlying data, which are based on past experience. Selection of variables for a model also reflects 
policy choices and value judgments. 

Another concern voiced by student advocates is that adjusting for educational outcomes, in particular, may set the 
stage for lowered expectations for certain groups of students or students at certain institutions. An adjustment 
model may be appropriate (and even necessary) for leveling the playing field for programs or institutions if 
outcomes are tied to funding. However, it may not be appropriate if the goal is to increase awareness and advance 
equity of outcomes . 5 

Conclusion 

Reliable, comprehensive information about the performance of postsecondary institutions is essential for student 
decision making and to promote improved access, progress, completion, and post-graduation results. CLASP 
supports increasing the transparency and accountability of postsecondary results and sees PIRS as an important 
instrument to achieve these goals. As PIRS evolves, it also needs to be implemented in a way that supports the 
needs of students from disadvantaged backgrounds. A well-designed PIRS should include results for a range of 
metrics, as described in this paper. It should disaggregate the results for these metrics for programs of study and for 
key demographic groups, so that students can see how institutions perform for these populations. A well-designed 
PIRS should also support the comparison of results for an institution with other institutions within its peer group. 
Moreover, it should provide earnings results for programs of study presented in their labor market context. 

Finally, PIRS should only be used for accountability purposes after substantial improvements have been achieved 
in the measurement of key metrics, as described in this paper, and with careful attention to the ways in which 
performance expectations are set. This will minimize the risk to the most vulnerable students. 


4 Measure Twice: The Impact on Graduation Rates of Serving Pell Grant Recipients, a Policy Bulletin for HEA Reauthorization, July 2013, Advisory 
Committee on Student Financial Assistance, Washington, DC. 

5 (See Burt S. Barnow and Carolyn J. Heinrich, One Standard Fits All? The Pros and Cons of Performance Standard Adjustments, 2009) 
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